See Singh et al. (2000). Moreover, SARSA and Expected SARSA are also both appropriate, if the agent is greedy in the limit. Note that condition 2 requires that the agent takes every action in every state infinitely many times Proof. Let A satisfy the following in a given NDP: A is greedy in the limit, i.e. for all δ > 0, P (Q A's Q-values are accurate in the limit, i.e. if π Then φ has a fixed point. Theorem 3. Every continuous NDP has a strongly ratifiable policy.

Newcomblike decision problems have been studied extensively in the decision theory literature, but they have so far been largely absent in the reinforcement learning literature.